Search Results

Visual Studio Code Extensions and Settings

I primarily work in Visual Studio 2017 and Visual Studio Code, using VS2017 for SSDT work, and VS Code for pretty much everything else. VS code is highly configurable, and as it’s a rainy Sunday, I thought I’d share my settings with you in case you are interested. A few colleagues at work have asked me what extensions and settings I have so, here they are as of Feb 2018. Extensions In a console session within vs code, you can do this to list them: PS C:\> code --list-extensions codezombiech.gitignore DotJoshJohnson.xml eamodio.gitlens gerane.Theme-Blackboard mohsen1.prettify-json ms-mssql.mssql ms-vscode.PowerShell ms-vsts.team PeterJausovec.vscode-docker secanis.jenkinsfile-support yzhang.markdown-all-in-one

Creating VSTS Extensions for Visual Studio Marketplace

Hello! One of the neat things about VSTS is the Visual Studio Marketplace. To quote the VSTS Support Page , the Visual Studio Marketplace is “new destination and the exclusive place for purchasing subscriptions, and for discovering extensions for Visual Studio Team Services and Visual Studio Code.” OK, sounds pretty exciting. But what is an Extension? Again, let’s use Microsoft’s own documentation about Extensions – “Extensions are simple add-ons that can be used to customize and extend your DevOps experience with VSTS. They are written with standard technologies - HTML, JavaScript, CSS - and can be developed using your preferred

Establishing a git workflow with VSTS and Visual Studio - git rebase

In this post I will look at git rebase in Visual Studio, in particular rebasing a working branch onto master and at what happens to each commit. This post will not make a case for rebase over alternatives, though the findings will contribute to the workflow that will be decided upon. All posts in this series on establishing a git workflow for VSTS and Visual Studio are: Establishing a git workflow with VSTS and Visual Studio Configuring a VSTS repo from the Github sample repo Git merge Git rebase (this post) Recap on setup If you have just landed on

Establishing a git workflow with VSTS and Visual Studio - git merge

In this post I will look at git merge in Visual Studio, in particular merging down from master into a branch and at what happens to each commit. This post will not make a case for merge over alternatives such as rebase, though the findings will contribute to the workflow that will be decided upon. All posts in this series on establishing a git workflow for VSTS and Visual Studio are: Establishing a git workflow with VSTS and Visual Studio Configuring a VSTS repo from the Github sample repo Git merge (this post) Git rebase Recap on setup If you

Establishing a git workflow with VSTS and Visual Studio - pull requests

In this post I will look at pull request processing in VSTS. Over and above the pull request process, what is also interesting is any differences in the commit graph on the remote repo when merging up from a merged as compared to a rebased branch. As with previous posts in this series, this post will not make a case for either, though findings will contribute to the workflow that will be decided upon. All posts in this series on establishing a git workflow for VSTS and Visual Studio are: Establishing a git workflow with VSTS and Visual Studio Configuring

Establishing a git workflow with VSTS and Visual Studio - configuring a VSTS repo from the Github sample repo

Mid-way through writing what I expected to be the second post on establishing a git workflow with VSTS and Visual Studio, I realised the second post needed to be about configuring a VSTS repo from the Github sample repo git_workflow_sample_setup , since this action will be repeated for each of the subsequent posts that will look into merge, rebase and commit squashing. The sample code and branch structure are described in the first post Establishing a git workflow with VSTS and Visual Studio . First of all Throughout this post I assume the reader has some familiarity with VSTS. When

Establishing a git workflow with VSTS and Visual Studio

Here at sabin.io we use VSTS backed by git repos. We have a wide range of ability in git and though some of us have being using git for several years, we are not git wizards. The this is git cartoon by the folks over at xkcd.com pretty much sums up my early experiences, which in turn enforced narrowing my use of git to a limited workflow and a preference of git bash over GUI. And so, even though I’ve been using git for several years, I have only scratched the surface. When tasked with defining our workflow, I found I

How To Use MSBuild Arguments in Visual Studio Builds

When creating MSBuild.proj files for builds in Visual Studio, there are times when you might want some targets called, and other times you don't want the targets called. In my case, our Production builds compile all the web and windows service solutions as well asthe deployment scripts for databases and our BIDS projects. Essentially everything I would want if I was going to run an entire deployment for a release. However when deploying a dll hotfix, I don't want the SQL scripts and BIDS stuff built. When we want to deploy a hotfix, speed is of the essence. Our build

VSTS Git Repos no longer show as connected in Visual Studio

I hit something this week where one of my Git repos showed as disconnected in Visual Studio and Team Explorer was showing me the option to clone the repo. Which clearly I didn’t want to do as it was already cloned. You can see in the picture that I have the repo locally (at the bottom) but in the list of projects I get the clone button.   The root cause of this was that VSTS has been cleaning up an old artefact of its TFS history. For on premise TFS you have the ability to have collections, and one

T SQL Tuesday: Shipping Database Changes with SSDT

Hello! Let’s see how this goes: this months subject for T SQL Tuesday is about shipping database changes, something we here are all familiar with. So I thought I’d make some notes about a tool I’m very familiar with, SQL Server Data Tools. The Good It’s free! SSDT works with Visual Studio Community up to Ultimate, and from Visual Studio 2015 onwards it comes with it’s own Visual Studio IDE. SSDT Has a NuGet package available. So you don’t need to install Visual Studio to get builds running, and crucially can control which version is used to compile at a

Automating SQL Server Performance Testing

You run performance tests as well as functional tests when deploying new code changes to SQL Server, right? Not many people do, I think you should, and this article will show you how to do it by harnessing an existing performance tool, rather than writing your own monitoring infrastructure from scratch. Any good performance monitoring tool that records information to a database will do fine, and we prefer to use Sentry One . Here are the steps to accomplish this. Create a baseline database When you release your database change, you want to have something to compare against as an

How To Compile SQLProj Files Using Cmdline MSBuild... Errors Included!

I recently needed to build and deploy about 40 small database projects that were in 4 or 5 different database solutions. And I needed to do this several times a day, so compiling via Visual Studio would be a boring and tedious process. So to speed up the process I decided to write the build process in an MSBuild target file and call initiate the build process through PowerShell. The targets file was simple enough to put together. This would be saved in the root location of all the solution folders as "BuildAllDBProjects.targets.xml". Then the PowerShell would be simple enough;

Feedback requests to Microsoft

If you didn’t know Microsoft has a number of channels to provide feedback. Most historically user connect (connect.microsoft.com), it integrated with their internal bug tracking systems and meant that items flowed from the users to engineering and back. Well supposed to.   The SQL product group still use connect https://connect.microsoft.com/sql with a few teams also using Trello https://trello.com/b/NEerYXUU/powershell-sql-client-tools-sqlps-ssms and or Slack Slack - sqlcommunity.slack.com Visual studio is moving to https://developercommunity.visualstudio.com/spaces/8/index.html from connect and also has https://visualstudio.uservoice.com/forums/121579-visual-studio-ide for ideas VSTS has a great support and also uses MSDN, and takes requests on Uservoice https://visualstudio.uservoice.com/forums/330519-team-services PowerBI has forums and uses user voice

Why is SSDT Always Rebuilding My Constraints?

Hello! Let me begin by saying that I’m a big fan of SSDT. It’s free, it works with all flavours of Visual Studio, the team do their very best to keep up-to-date with features that are released with increasing regularity by Microsfot in both Azure and SQl Server on-prem. I’ve met the team a few times, and they’re genuinely keen to engage with the users of SSDT in how it can be better, and how it can be extensible. So, SSDT is a great tool. I said great, but not perfect. It has it's limitations, the same as any tool.

Keeping The Database Dev Ops Overhead Lightweight

Hello! One very important aspect of Dev Ops that is perhaps over-looked is the overhead that comes with adopting Dev Ops practices. To help explain what I mean, let’s break that sentence down a bit. What Do I Mean By “Dev Ops Practices” I have a strong suspicion that for each of the posts for this T-SQL Tuesday on Database Dev Ops, everyone will have a slightly different take. Or rather, they are going to articulate what Dev Ops means to them. And so here is my take: broadly speaking, Dev Ops is about increasing the cadence of a feature

SSDT 16.5 Released

Hello! Recently the SQL Tools Team released a new version of both SQL Server Data Tools ( SSDT ) and SQL Server Management Studio (SSMS.)  There’s a range of bug fixes, but two new features that I am particualrly interested in. Firstly, SQLPAckage.exe and the DacFx API can now generate deployment report, deployment script and publish to a datbase all in one action. Neat! This is useful because it’s important to keep track of exactly what has changed on a database. Of course there’s nothing stopping you right now from creating executin these options in three separate actions, but there

When to use CmdletBinding in PowerShell?

Clean Code At Sabin.IO we are big proponents of clean code. We use PowerShell a lot for automation, and want our code to be clean. You are automating everything, right? If not, please see a slide from a recent meetup: For me, clean code in PowerShell means (and not limited to): Small self-contained functions that have a single responsibility Number of arguments to a function kept as small as possible Consistent formatting No duplication of code Modules that hide internal functions, and only expose what’s needed Common Parameters One way to make code a bit cleaner is to make use

ARM Template Snippet - Allow Azure Resources To Access Azure SQL Instance With This One Simple Trick!

  Hello! I’m really getting into using ARM Templates to deploy Azure Resources. For those of you not ITK, and for the sake of brevity I’ll summarise in a sentence, ARM Templates are basically Infrastructure as Code. It’s all very interesting, go and have a read . In addition to having our code in source, the ability for us to deploy our entire infrastructure from source into Azure is fantastic, because now we can run one single script to build deploy the infrastructure and the code from source, and we can all have our own personal environment in Azure. And

Uploading Files To Data Lake Store With PowerShell Part One

Hello!   I’ve recently been working on uploading files to Azure Data Lake Store . It’s quite straightforward and I think a decent introduction into automating a deployment with Azure, as well as a good example of writing scripts that are idempotent, so I’m going to go through them from beginning to end. I’m going to go into one function per day, so this will take 5 days to cover. But I’m hoping that by focusing a bit more in-depth as opposed to trying to cram it all into one post it will be more informative, and both yourselves and

Continuous Integration with Jenkins, SQL Server and Windows Containers

Why use Windows Containers? When creating database applications we need consistency in all our environments to ensure quality releases. Traditionally developers might have their own instance of SQL Server on their workstation to develop against. Database projects would be created in SSDT and pushed to source control when ready for testing. If you’re not using SSDT for database development already, then you should seriously consider it to make your life easier and increase the quality of your releases. Ed Elliot explains why in this blog post . A problem with CI for databases is that databases are a shared resource

Do Azure SQL Database External Tables have a place in a micro-service?

I was recently in discussions on using External Tables to link Azure SQL Databases across micro-service boundaries. This has led to some challenging discussions with a client and unexpected opinions internally here at sabin.io . My simple view of a micro-service is of a data store fronted by code, which is in turn behind an API or message subscriber. Importantly only this code accesses the store. I have arrived at this opinion though many (often heated) discussions with developers implementing services, and though working with teams breaking large services into micro-services to clarify ownership and responsibility, remove dependencies and simplify

Microsoft Release SQL Server 2016 SP1 and Give Stuff Away!

Today was the Connection (); 2016 Keynote, and it was certainly was worth tuning into to catch-up with what Microsoft have been up to. I was going to write a brief summary of the Keynote and made some notes in a text file, but an hour in to the 2 hour presentation I gave up because there was so much going on. Microsoft have been pushing for an Incremental Service Modelling , to the point that they changed the recommendation from Cumulative Updates being deployed only if you are affected by any of the issues to always update . This

When To Use Octopus Deploy Script Modules

Hello! Lately I've been thinking a lot about script modules in Octopus Deploy. Script Modules, for those of you not ITK, are a collection of PowerShell cmdlets or functions that exist outside of a script step that can be used across multiple steps across different projects/environments. They don't have to be used to contain modules exclusively; they can just contain that will get executed upon the initial import of the module. The issue here is that these script modules are imported or every step in a process. So unless the code you write is idempotent, you'll probably cause an error

Webinar: 8th Feb 2018

Preventing SQL Server Performance Problems Before They Hit Production Join Mark Allison, Paul Anderton and Kevin Kline at 3pm UTC on 8th Feb 2018 Wouldn’t it be great if performance problems in your SQL estate could be detected BEFORE they reach your production databases? In this demo-centred webinar we will review: How to detect and prevent releases of code that could reduce performance of your SQL Server database Ways to prevent the most common performance problems before they reach production: missing indexes, deadlocks and excessive key lookups How effective SentryOne can be in a DevOps pipeline both on-premise and in

Fix For Using Azure Active Directory and DacFX

Hello! As part of an SSDT project we have a contained user that authenticates against an Azure Active Directory group (read more on the CREATE USER page). However the account we are executing deployments with is the SQL Admin account on the Azure SQL Instance. And so we get this error - The executed script: CREATE USER [myUser] FOR EXTERNAL PROVIDER; ' Reason: '' At line:94 char:13 + Throw $toThrow + ~~~~~~~~~~~~~~ + CategoryInfo : OperationStopped: (Deployment fail... ' Reason: '':String) [], RuntimeException + FullyQualifiedErrorId : Deployment failed: 'Could not deploy package. Error SQL72014: .Net SqlClient Data Provider: Msg 33159,

SSDTPokedex: Migrating a Database Into SSDT

Hello!   If you want to have the best chance of something being successful, you have to be committed to it from the start. That’s a pretty fatuous sounding statement, almost as bad as “to make something better you have to do more of the good stuff and less of the bad stuff”, so let me contextualise: if you desire to have good testing coverage on an application, then you need to be serious about testing form the first day you write code for the application. Be it manual or automated testing, you need to put the effort in terms

Are you running VMware ESX 6 Update 1 and connecting to SQL - be warned

If you are running any of your application on windows 2012 and up and on VMware ESX 6 with Update 1 connecting to SQL you must read on. We’ve been working at a client recently on an issue where requests to SQL have a 500ms latency between requests. What’s really odd is that the time seems to be lost between the client code making a request and the request coming to SQL. Profiler shows a gap of 500ms between the end time of one request and the start time of another request. The issue would only affect some applications and

Git: How to work on a feature branch but pull in later commits from another branch?

On a client site last week the question was raised: I want to work on a feature for a project which will take longer than other people merging their branches into the dev branch. I therefore want to merge any changes on the dev branch into mine, test my changes before pushing back to the dev branch, and then and ultimately master for a release. We are using a git flow methodology whereby the development manager will merge feature branches from developers’ branches via pull requests into the dev branch. Tests are run and if they pass, we merge into

Running SQL Server in an Azure Container Instance

Azure Container Instances are still in Preview and not officially available for Windows yet, which made me smile. It took me a while to figure out how to get this working so I thought I’d share what I’ve found. Containers are great for lightweight testing of code before deployment to production servers because they can be created so quickly and they give the same environment to test in very reliably. Now that Microsoft is offering container instances in Azure it means you don’t have to worry about provisioning and configuring your own docker host/cluster. The options for deploying SQL Server

Migrating SSIS Packages to SSIS Azure part Two – Automating the Deployment

Hello! If you’ve read and followed through my previous post, you will have World Wide Importers Integration Services project running in SSIS Azure. It’s all very interesting, go and have a read . One thing that is missing form that guide, the documentation, and SSIS in general, is how to automate SSIS Deployments. In the WWI SSIS project, there are connection managers that we had to manually update the values of to get it to work post-deploy. This is exactly the opposite of what we want to do. Back when SQL Server 2012 was known as Denali, one of the

VMWare network performance bug - Getting a repro

If you’ve read my previous post about an issue with VMware ESX 6 and connecting to SQL and 500ms latency , you might be interested in the process we went through to get to the repro. Getting a repro (being able to reproduce a bug/feature) is often a complex and time consuming task. The challenge is like being Sherlock Holmes and using your experience to focus on the aspects of the situation that is important. The challenge is that without a repro, You can’t give anything to a supplier to enable them to triage and find a fix for it

Migrating SSIS Packages to SSIS Azure

Hello! In case you missed the announcement (and there were a lot of announcements during MSIgnite), SQL Server Integration Services is in Public Preview on Azure! I’ve written about it elsewhere in greater depth , but here are the headlines: It makes use of SSIS Scale Out , which was released as part of SQL Server 2017 . Although it is based on SSIS Scale Out, you can’t actually configure SSIS Scale Out to run on the instance. If this confuses you then read my in-depth post. SSISDB is installed in either SQL Azure or on a Managed Instance. You

SSDTPokedex: Integrating Slack and VSTS Into GitHub Repo - An Infinite Improvement

Hello! One of the home projects I’m currently working on is migrating a database over from SQLite to SQL Server. There’s several tasks that need to be accomplished before we can say that this is successful. Broadly speaking they fit into the key pillars of successful software development: Plan Develop Deploy Measure   So the development part is well under way: there is a repo in GitHub of an SSDT solution that will compile locally (it works on my machine anyway.) Plus I have a couple of releases: one intentionally broken and one fixed. Now there’s plenty of tasks I

VSTS Hosted Build Agent Specs

I was interested to know just what the hardware specifications of the hosted build agent is. So I added some PowerShell to read out the info below: 2016-06-29T09:23:31.3935358Z systemname      Name                                      DeviceID NumberOfCores NumberOfLogicalProcessors Addresswidth 2016-06-29T09:23:31.3935358Z ----------      ----                                      -------- ------------- ------------------------- ------------ 2016-06-29T09:23:31.3935358Z TASKAGENT5-0010 Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz CPU0                 2                         2           64 2016-06-29T09:23:31.4095356Z Total memory:  7167.55078125 What piqued my interest greater was that this is the exact same spec for a D2 v2 box that is available via Azure. Clearly, Microsoft have a build agent template which is built, stored in a pool, and provisioned whenever a build

SSDT and Pre-Model Scripts

Hello! pre-deploy scripts in SSDT scripts may be executed before the main diff script is executed, but not before the diff script is generated. This is an important fact to understand . If you want scripts to be executed before the diff is generated, then you need to execute pre-model scripts. How you go about this is up to you, but there needs to be a few rules that must be followed: Idempotent : big word, and is used in maths. But don't let that put you off. What it means in this context is that a script can be

CosmosDb, know your costs, and remember…

This will be a short post to emphasize a simple point, yet one that should make an enormous difference to how you approach configuring a CosmosDb collection and modelling documents to support read and write requirements. Know your costs I cannot emphasize this point enough. The folks at Microsoft have made this really easy, be it via the Request Units (RU) and Data Storage calculator , the collection Query Explorer through the Azure Portal or a REST client such as Postman coupled with the really useful library and samples by a Microsoftie over on git documentdb postman collection . Let’s

Automatic Tuning Enabled By Default In Azure SQL Database Happens Today!

Hello! From January 15th (ie, this Monday) Automatic Tuning will be enabled by default and gradually rolled out to ALL Azure subscriptions. If you are the owner of a subscriber you would have received an email two weeks ago alerting you of this fact. However if you’re not, and this has not been communicated out to you, this may be something of a surprise. With regards to how the rollout impacts you, this blog post states that “All servers that do not have automatic tuning explicitly configured will inherit Azure defaults, making automatic tuning enabled. Similarly, all databases that do

Running a SQL Server workload using PowerShell

In February 2018, myself and Paul Anderton gave a presentation on how to correlate database deployments with performance issues within the context of a DevOps pipeline. We used Sentry One as our monitoring tool in a Performance Test environment so that we could catch badly performing deployments before they got to production and caused havoc. If you would like to see the recorded video, then you can download it from here: http://info.sentryone.com/partner-webinar-performance-problems As part of this presentation we had a workload running on a workstation, which executed a couple of stored procedures repeatedly, and we’ve had some requests from people

PowerQuery – The power of M

I love PowerBI, actually I love PowerQuery. It's a great way to combine data from around your business, across the web from anywhere. What I really like is very little is done automatically, i.e. it doesn't do the nice data type detection you get with Excel that screws your imports if the data type in a column differs from the first few rows to the rest of the file. Does that make it difficult to use. No its not. The nice thing is that its very easy to add additional columns, change data types, use expressions, combine datasets, and do

How to install SQL Server on Windows Server Core?

As part of automation of database and application deployments, it makes sense to be able to create new SQL Server instances quickly and with minimal resources. I have already explored containers and written about it on this blog, but I’d like to turn your attention to setting up SQL Server on Windows Server Core for those of you that run SQL Server on-premise or within VMs in the cloud. In a domain environment it should be pretty simple to just create a PowerShell session to your target Windows Server where your account is a local administrator and then simply run

Team City Meta Runner - Get Build Status

Hello! When building a deployment pipeline, the choice of tool is less important than the use of the tool: do you go for a tool that centrally controls the flow of a release, from build to running tests to actual deployment, or do you choose separate tools that are loosely hung together and execute a particular part of a release? From personal experience, I have preferred to use a tool to act as a control flow of the deployment pipeline, but leverage tools where there is clear sense to use them. A case in point is using TeamCity to run

CosmosDb, know your partition costs, well more or less

In my previous post Cosmos Db know your costs, and remember I made the point that by understanding RU costs early, you can make informed decisions in relation to document design and application CRUD and query operations. While it is easy and most certainly useful to arrive at a projected RU cost, using for example, the Request Units (RU) and Data Storage calculator or directly against a fixed 10GB collection via the Azure Portal (incidentally the same costs), the problem is these do not highlight RU costs when partitioning is required to support scale-out . Now if you know your

Managing Azure Functions logging to Application Insights

The Azure Functions teams have made it incredibly easy to emit telemetry to Application Insights. It really is as easy as update the Function App’s settings as described by the App Insights wiki page over at Azure Functions on github. However, if you are on the basic pricing plan for Application Insights then the 32.3Mb daily allowance gets used up pretty quickly. The remainder of this post is about understanding the telemetry data sent to Application Insights by Azure Functions and how to configure the function app host.json to filter and reduce the volume of telemetry sent. The naïve approach

How to check Sentry One requirements

I was at a client site recently and implemented Sentry One for them, a great monitoring system for SQL Server. It proved challenging because some servers were in a DMZ on a separate network and domain and some servers were in the same domain. All servers connected via a router and were firewalled off from each other with only the minimum ports open required for them to fully function and communicate. Sentry One operates in two modes, Full and Limited. Full mode allows Sentry One to gather Windows metrics as well as SQL Server metrics Limited mode does not allow

Improving Azure Functions throughput

I recently ran into an Azure Functions throughput problem which I logged on Stack Overflow as regular-throughput-troughs-in-azure-functions-requests-per-second . The product group were pretty quick to respond and pointed me to their Azure App Service Team Blog post processing-100000-events-per-second-on-azure-functions/ . The post lists five notable configuration choices: functions process [event hubs] messages in batches webJobs dashboard is disabled in favor of using Application Insights for monitoring and telemetry each event hub is configured with 100 partitions data is sent to the event hubs without partition keys events are serialized using protocol buffers Of these, the second and third are most interesting.

CosmosDb and CRUD with a S

In preparation for my talk on Azure SQL Database vs. CosmosDb - How do you choose   at giving a talk at sqlbits I spent a lot of time thinking about create, read, update and delete (CRUD) operations, and concluded that with the advent of CosmosDb and its multi-model abstraction layer, CRUD needs a S. CRUD as an acronym has served software engineering very well by encapsulating the core actions of data persistence and retrieval. In the transition from the “ye olde” closed and monolithic systems to the current open, loosely coupled and distributed micro-service architectures via client server and

Assist Deploy Is Available on GitHub

Hello! For some time now I have been working on automating SSIS deployments, and earlier this week I published my efforts on GitHub . But before I get into the what/how, let’s focus on the why and let me catch you up on how I got here… The task to take an ispac and deploy in and of itself is quite a straightforward process as there are multiple ways to do this . For those of you who want the abridged version of the linked post, the choices are as follows: Integration Services Deploy Wizard SSIS Catalog T-SQL API PowerShell

SQL Server Container Performance

Is SQL Server in a container faster than a VM? I briefly looked at SQL Server containers when Windows Server 2016 was released. Containers offer the ability for rapid provisioning, and denser utilization of hardware because the container shares the base OS’s kernel. There is not a need for a Hyper-Visor layer in between. As a recap for those that are not up speed with containers, the traditional architecture of databases in a VM is like so: The Hyper-Visor OS is installed onto the host hardware, a physical server in the data centre. Many VMs are created on the Hyper-Visor

SSIS Package Execution In Azure Is Now Available

Well, it’s been some time coming but SSIS packages are the latest product to make the move from on premise to Azure. You can now take your SSIS projects and deploy them to the new Platform as a Service (PaaS) offering in Azure. The aim of the team at Microsoft was for users to take their current SSIS packages and just “lift and shift” these to Azure. So in development terms that means that there are minimum to no changes to be made in the solution at least. But before we get into the deployment and running of SSIS packages